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Most research on school eflfectiveness has focused solely on academic outcomes 
among pupils. In standardised educational systems, such as England and Scotland, 
academic outcomes tend to be measured in terms of examination results (see, for 
example, Goldstein et al., 1993). In other systems, such as the United States, 
outcomes are measured in terms of standardised ability test scores (see, for example, 
Teddlie and Stringfield, 1993). In comparison, studies which focus on both academic 
and non-academic outcomes among pupils are comparatively rare. Some studies have 
considered pupil behaviour (Rutter et al., 1979; Mortimore et al., 1988) and/or 
personal/social development (such as academic self-image and locus of control) among 
pupils (see Brookover et al., 1979; Bryk et al., 1993). However, findings have differed 
concerning the relationship between academic and non-academic outcomes. 
Mortimore’ s study indicated no discernible relationship between school effects in 
relation to cognitive and non-cognitive outcomes (see also Brookover et al., 1979), 
while Rutter’s study indicated that schools which did better in terms of exam 
performance also did better in terms of pupil behaviour and delinquency. More 
recently, researchers have stressed the need to study a broad range of educational and 
developmental outcomes of the schooling process (Knuver and Brandsma, 1989; 
Creemers and Scheerens, 1994; Gray, 1995). 

This paper uses data from a national survey of second-level schools in the 
Republic of Ireland to assess the relationships among, and the factors influencing, a 
range of pupil outcomes, both academic and non-academic. The analyses relate to 
pupils taking the Junior Certificate examination, a nationally standardised examination 
taken at 15-16 years of age. 
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Methodology 

A national survey of second-level schools and pupils in Ireland was conducted in 1994. 
A sample of 116 schools was drawn using stratified random sampling with schools 
selected to be representative of the national distribution of schools in terms of school 
sector, gender composition, school size and location. Detailed interviews were carried 
out with school principals and guidance counsellors in order to obtain information on 
key aspects of school organisation and process. The pupils selected for study were the 
Junior Certificate and Leaving Certificate exam year groups. Classes were sampled 
within schools, taking roughly half the total number of classes from the relevant years 
in each selected school and allowing for differences between schools in the system of 
class allocation (streaming as opposed to mixed ability) in selection. Pupils at Junior 
Certificate (15-16 years of age) and Leaving Certificate (17-18 years of age) 
completed questionnaires which covered topics such as take-up of subjects and levels, 
family background of the pupil, attitudes to school, perceptions of school climate and 
interaction with teachers, pupil stress levels and other aspects of personal/ social 
development. In total, questionnaires were completed by 5,961 Junior Certificate and 
4,813 Leaving Certificate pupils (see Hannan, Smyth et al, 1996). 

For the Junior Certificate group, verbal reasoning and numerical ability (VRNA) 
tests were administered approximately three months before the Junior Certificate 
exams. In addition, information was obtained from teachers on pupils’ attendance 
record over the academic year. It would have been preferable to have intake measures 
of pupil ability in order to explore the impact of school characteristics on subsequent 
performance. However, previous analyses (see Hannan, Smyth et al, 1996) indicated 
that third-year test scores could be taken as a broadly reliable proxy of differences in 
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ability on intake*. Data were subsequently added on pupils’ exam grades in the Junior 
and Leaving Certificate exams. A second phase of the study involved detailed case- 
studies of six of the 116 schools; these case-studies explored perspectives among 
school management and teachers in order to explore the complex interaction of school 
organisation ‘on the ground’. However, analyses in this paper primarily relate to the 
larger sample of schools rather than to the case-study material (see Smyth, 1999, for 
further details). 

A number of measures of pupil outcomes are used in this paper: 

• Examination performance at Junior Certificate level, measured using the average 
grade (GPAV) received per exam subject; 

• Pupil absenteeism, measured using teachers’ reports of attendance records among 
Junior Certificate pupils; 

• Potential drop-out among pupils, measured using pupils’ reports of their intentions 
after the Junior Certificate exam; this information was supplemented by school 
records on actual rates of pupil drop-out over the junior and senior cycles; 

• Current stress levels among pupils, measured using an adapted form of the 
conventionally used General Health Questiormaire scale (see Hannan, O Riain, 
1993); 

• Academic self-image among pupils, a measure of how pupils evaluate their own 
academic abilities; 

• Locus of control among pupils, a measure of the extent to which pupils believe 
they are in control of events; 
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'T his conclusion was based on analyses of the twenty-one schools in the sample for which 
intake measures were available. 
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• Body image among pupils, that is, pupils’ evaluations of the attractiveness of their 
own bodies and self-presentation to others. 

Further details on the construction of these variables are given in Appendix 1 . 

Analyses presented in the paper used the MlWin package for multi-level 
modelling developed by the Institute of Education, University of London (Rasbash et 
al, 1999). Two sets of analyses were conducted. Firstly, multivariate multilevel 
modelling techniques were used to model the seven pupil outcomes as functions of 
pupil background variables. This facilitated an exploration of the relationships among 
different dimensions of school effectiveness, controlling for differences between 
schools in pupil intake. Secondly, analysis of a restricted set of school-level factors was 
carried out in order to assess whether different aspects of school organisation and 
process were differentially associated with various pupil outcomes. As multivariate 
multilevel analyses are computationally intensive, it was not possible to model all seven 
outcomes simultaneously as functions of both pupil-level and school-level factors. 
However, the second part of the paper brings together separate analyses of each pupil 
outcome in order to assess differences between them in their influences. 

School difTerences in pupil outcomes 

Figure 1 indicates the extent of variation among schools in a range of pupil outcomes, 
both academic and non-academic. The intra-school correlation indicates the amount of 
variation in a particular outcome which is attributable to the school, before controlling 
for pupil composition or school characteristics. It is Clear that the greatest difference 
between schools is found in relation to academic performance. In contrast, only two to 
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five per cent of the total variation in pupil development outcomes is attributable to the 



school level.^ 



Figure 1: School Variation in Pupil Outcomes 
(intra-school correlation) 



0.25 
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This pattern is not altogether surprising. One of the primary purposes of schools 
is to foster intellectual development among its pupils and examination performance 
represents a potential, though far from perfect, measure of such development. In 
contrast, few would argue that the enhancement of body image should represent an 
explicit objective of the schooling process. Thus, schools are more likely to influence 
the outcomes on which they focus more explicitly.^ In addition, personal-social 
development among pupils is likely to be subject to a very broad range of influences, 
including family circumstances, neighbourhood effects, peer groups (outside school) 



^ As absenteeism and pupil drop-out are binary outcomes, the proportion of variation 
attributable to the school level cannot be derived in the same way. 

^ It is noteworthy, however, that interviews with teachers and school management indicate that 
they tend to adopt a complex view of the objectives of schooling (see Smyth, 1999). 
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and so on, It is not unusual, therefore, that there are considerable differences in pupil 
development among those in the same school. 

It is evident that schools make more of a difference to academic outcomes, 
such as performance, than to personal/social development among pupils. However, it 
would also be interesting to determine whether school effectiveness can be regarded as 
uni-dimensional, that is, whether schools that are effective in academic terms are 
equally effective in terms of pupil development. The following section of this paper 
explores the extent to which these outcomes are inter-related at the school and pupil 
level. 

Pupil outcomes at Junior Certificate level 

Tables 1 and 2 present multivariate multilevel models where the seven pupil outcomes 
are modelled simultaneously as functions of the pupil background variables. This is 
carried out in terms of a three level model with schools treated as level 3 units, pupils 
as level 2 units and the ‘within student’ measurements (the seven pupil outcomes) as 
level 1 units. The model is a mixed response model since some of the outcomes are 
continuous and some are discrete (i.e., absenteeism and potential drop-out). A 
multivariate approach maximises the use of available data since there is no requirement 
that information be available for each pupil on all seven of the outcomes. The 
advantage of using a multivariate approach is that it allows us to examine the 
interrelationships among the outcomes at both the school and pupil level. 

Table 1 presents the null model which indicates the school- and pupil- level 
variation before allowing for differences between schools in pupil intake. The school- 
level variances (shown on the diagonal) indicate that schools differ significantly from 
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each other in exam performance, absenteeism, potential drop-out, stress, academic 
self-image, sense (locus) of control and body image at Junior Certificate level. 

In general, academic outcomes are significantly interrelated at school level; 
schools in which average exam performance is higher have significantly lower levels of 
absenteeism and drop-out. There are also some significant relationships between 
academic and non-academic outcomes. Schools with higher levels of pupil 
performance also tend to be characterised by more positive academic self-image and 
greater locus of control. However, schools with higher performance levels also have 
higher stress levels, a finding which indicates that there may be some 'trade-off with 
higher performance being achieved at the expense of greater stress among pupils. 
There are also significant relationships among some measures of personal/social 
development; schools in which academic self-image is higher also tend to have higher 
average locus of control and body image. 

Academic outcomes are also interrelated at the pupil level. Pupils who perform 
well in the Junior Certificate exam are more likely to have good attendance records 
and less likely to intend to leave school early. Measures of personal/social development 
are correlated with each other; pupils with higher academic self-image have more 
positive feelings of control and body image. Both academic self-image and locus of 
control are positively associated with pupil performance, although higher-performing 
pupils tend to have higher stress levels and more negative body images. 

Table 1 shows the 'raw' relationships between the different pupil outcomes. The 
relationships among outcomes may be spurious, however; for example, the positive 
relationship between exam performance and stress at the school level may reflect a 
greater concentration of girls (who have higher exam scores and higher stress levels) in 
certain schools rather than reflecting a school effect per se. The model presented in 
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Table 2 includes pupil background and ability in order to allow for differences between 
schools in pupil intake. 

There are clear differences among pupils in terms Of their gender, social 
background and age. Girls appear to be at a relative advantage in terms of academic 
outcomes, having higher exam scores, lower potential drop-out and somewhat lower 
absenteeism rates, but are at a disadvantage in terms of measures of personal 
development. Girls tend to have higher stress levels, less sense of control over their 
lives and make more negative evaluations of their abilities and appearances than their 
male counterparts (Table 2). 

Working-class pupils tend to be at a relative disadvantage in relation to both 
academic and non-academic outcomes; they have lower exam scores and are at greater 
risk of absenteeism and early school leaving, relative to their initial ability levels, than 
their middle-class counterparts. In addition, working-class pupils have less sense of 
control over their lives and more negative evaluations of themselves. A similar pattern 
is found in relation to mother’s education, with higher performance, lower absenteeism 
and drop-out and more positive academic self-images found among pupils whose 
mothers have higher levels of education. Pupils who are older than average tend to do 
worse academically than younger pupils; they have lower exam scores, higher 
absenteeism rates and are more likely to report intending to leave school early. This 
group is likely to contain a disproportionate number of pupils who had experienced 
grade retention during their time in primary or second-level school and thus the pattern 
is likely to reflect a longer-term process of educational under-achievement. As might 
be expected, higher 'ability' pupils tend to do better in the Junior Certificate exam. 
They also tend to have lower absenteeism rates and are less likely to intend to drop out 
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of school. 



In general, there are fewer significant interrelationships among pupil outcomes, 
especially at school level, when pupil background and prior ability are taken into 
account. At school level, academic outcomes are significantly interrelated with higher 
performing schools having lower absenteeism and drop-out rates, even when pupil 
background and ability are taken into account (Table 2). The reduced size of the 
effects compared to the ‘raw’ correlations, however, indicates that many schools 
deviate from this pattern. Some schools may, for example, enhance pupil performance 
while at the same time failing to reduce pupil drop-out. Measures of personal/social 
development are not strongly interrelated at school level. However, schools that 
promote academic self-image also tend to promote locus of control among pupils. 
There are no longer any significant interrelationships between academic and non- 
academic outcomes (with the exception of the surprising positive relationship between 
performance and body-image) so schools that promote pupil performance do not 
necessarily have positive effects on pupil development. In addition, the positive 
relationship between average performance and stress levels evident from the 'raw' 
results is no longer evident. There is, therefore, no necessary 'trade-off between 
academic effectiveness and raised stress levels among pupils. 

At the pupil level, the relationships among outcome measures are broadly similar, 
even when adjustments are made for pupil background and ability. Pupils who do well 
in the Junior Certificate tend to have other positive academic and non-academic 
outcomes (with the exception of body image), although they have higher stress levels. 




School factors 

Due to computing limitations, it was not possible to model all seven pupil outcomes 
simultaneously as a function of both pupil-level and school-level factors. As a result, 
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'side by side' analyses were carried out in which the results of a limited set of school 
factors were compared across the seven outcomes. A wider set of school-level factors 
has been considered elsewhere (Smyth, 1999). 

The first set of variables relate to the system of ability grouping used in the 
sample of schools. Three systems of allocating pupils to base classes were used: (i) 
streaniing, where pupils of similar assessed ability are grouped into classes, ranked 
from 'top' to 'bottom'; (ii) banding, a looser form of streaming with pupils divided into 
broad ability bands with mixed ability base classes within each band; and (iii) mixed 
ability, usually based on random (e.g. alphabetical) allocation or, more rarely, schools 
use ability measures to achieve a mix across classes. At Junior Certificate level, 37 per 
cent of schools use mixed ability base classes with the remainder using some form of 
ability-based differentiation (that is, streaming or banding). Within Irish schools, ability 
grouping has implications both for the range of subjects a pupil takes or is offered, and 
for the level at which a subject is taken. 

Figure 2 presents estimates of pupil performance in top, middle and bottom 
classes within streamed schools compared with pupils in mixed ability base classes, 
controlling for pupil background and 'ability'. It indicates significant underperformance 
among those allocated to bottom classes within streamed schools. However, there is 
no corresponding gain for those allocated to top classes; this group do not significantly 
outperform those allocated to mixed ability base classes. This pattern appears to reflect 
two processes. Firstly, pupils in bottom classes are often allocated to lower levels 
within their exam subjects; as a result, a 'ceiling' is set to their potential performance. 
Secondly, there appears to be a labelling process whereby pupils (and their teachers) 
have lower expectations within bottom classes. This alienation from the school is also 



manifest in the higher potential drop-out rates found among those in bottom classes 
(see Table 3). 

Figure 2: Exam performance by ability group 
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Disciplinary climate is significantly associated with higher exam performance 
and lower pupil drop-out rates. However, the effect is mediated by the academic 
climate of the school. In other words, pupils tend to perform better, and are likely to 
stay on, in schools characterised as strict but this mainly reflects higher expectations 
among teachers in the school. 

The quality of teacher-pupil interaction (from the perspective of the pupils) is 
significantly associated with all of the pupil outcomes, both academic and non- 
academic. Pupils who experience positive interaction with teachers (in the form of 
praise or positive feedback) tend to have higher exam performance, better attendance 
records, lower drop-out rates, lower stress levels, more positive self-evaluations and 
greater sense of control over their lives. In contrast, pupils who report negative 
interaction with teachers (that is, they feel they are frequently reprimanded or ignored) 
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have lower performance, higher absenteeism and drop-out, higher stress levels and 
more negative self-evaluations than other pupils. 

Academic climate reflects the extent to which teachers within the school hold 
high expectations for their pupils (see Appendix 1). Higher teacher expectations are 
associated with higher exam performance, lower absenteeism and drop-out rates 
among pupils. Interestingly, however, pupils in schools characterised by a strong 
academic climate tend to have slightly higher stress levels and more negative 
evaluations of their own abilities. This is likely to reflect somewhat greater exam 
pressure within these schools (see Smyth, Harman, et al., 1996). 

In summary, a number of factors, including positive teacher-pupil interaction and 
a stronger academic climate within the school, are associated with improved academic 
outcomes in the form of higher exam performance, better attendance records and 
lower pupil drop-out rates. The nature of pupil-teacher interaction within the school is 
significantly associated with both non-academic and academic outcomes among pupils, 
indicating the importance of informal school climate as well as formal school 
organisation. 

Conclusions 

Analyses in this chapter indicate that 'school effectiveness' must be seen as outcome- 
specific since schools that promote academic progress among pupils do not necessarily 
enhance their personal/social development. There is some indication that a dimension 
of academic effectiveness can be identified, that is, higher-performing schools tend to 
have lower absenteeism and drop-out rates. Similarly, there is some consistency among 
non-academic outcomes with significant interrelationships evident among academic 
self-image, body image and locus of control. Analyses at the pupil level indicate that 



pupils tend to do better in their exams when they have a better attendance record, 
intend to stay on at school and have a more positive image of themselves. 

It is debatable whether schools can be expected to significantly enhance all 
aspects of personal-social development among young people. Some developmental 
outcomes (such as positive body image) are not an explicit goal of the schooling 
process. However, even in these cases, schools may represent potential sites for 
intervention or referral for pupils with particular problems. Other outcomes, such as 
academic self-image, are relative in nature, that is, pupils assess their abilities in 
reference to their peers. Consequently, academic self-image cannot be equally high for 
all pupils in the school. However, if academic self-image is found to have ari impact on 
actual performance, low self-image among particular groups of pupils must be a matter 
for concern. 

In conclusion, the promotion of non-academic, as well as academic, outcomes by 
schools may be seen as a desirable goal. Analyses indicate that, while between-school 
variation in developmental outcomes is relatively small, certain aspects of school 
organisation and process can enhance personal/social development among pupils. 
Furthermore, attention within schools to the promotion of “non-academic” outcomes 
may in fact have positive effects on academic performance among individual pupils. In 
general, a consideration of a broader range of pupil outcomes than academic 
performance would appear to yield greater insight into the complex set of tasks and 
goals faced by second-level schools. 
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Table 1: Multivariate Model of Pupil Outcomes at Junior Certificate: Null Model 
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Table 3: Single-outcome models of school-level factors and pupil outcomes, controlling for pupil characteristics 
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Appendix 1: Derivation of Variables 


Variables 


Description 



Pupil outcomes 

Junior Certificate exam performance Exam grades are assigned scores from 0 to 10 and averaged 





over all exam subjects taken. 


High absenteeism 


Dummy variable where 1= Pupil has poor or average 
attendance records based on teacher reports. 


Potential drop-out 


Dummy variable where 1= Pupil is not definite about staying 
on to the Leaving Certificate. 


Stress 


Likert scale based on the following items: 

(1) Been able to concentrate on whatever you’re doing 

(2) Felt that you were playing a moderately useful part in 
things 

(3) Felt capable of making decisions about things 

(4) Lost much sleep over worry 

(5) Felt constantly under strain 

(6) Been losing confidence in yourself 
Reliability: alpha of 0.72. 

Original scores range from 6 to 24; scores were averaged 
over the six items to give a variable with a range of 1 to 4. 


Academic Self-Image 


Likert scale based on the following items: 

(1) 1 can do just about anything I set my mind to 

(2) I’m usually well ahead of others in my year in school 

(3) I am as good at school work as most other people my age 

(4) I’m hardly ever able to do what my teachers expect of me 
(reversed) 

(5) I’m usually well ahead of others in my class. 

Reliability: alpha of 0.67. 

Scores range from 1 to 4. 


Locus of Control 


Likert scale based on the following items: 

(1) 1 have little control over the things that happen to me 
(reversed) 

(2) There is a lot I can do to change my life if I really want 
to 

(3) I often feel helpless in trying to deal with the problems I 
have (reversed) 

(4) What happens in the future really depends on me 

(5) I can do just about anything I set my mind to 

(6) There is really no way I can solve some of the problems I 
have. 

Reliability: alpha of 0.50. 

Scores range from 1 to 4. 



Body Image 



Likert scale based on the pupils’ selection from adjective 
pairs; 

(1) Plain - Good-looking 

(2) Fat - Thin 

(3) Awkward - Graceful 

(4) Unattractive - Attractive. 

Reliability: alpha of 0.64. 

Scores range from 1 to 7. 



Explanatory variables 

Gender Dummy variable where 1= Girl. 

Social class Census Social Class scale ranging from 0 (Higher 

Professional) to 5 (Unskilled manual worker) based on the 
occupational status of parents. 

Mother’s education Highest level of mother’s education ranging from 0 (primary 

education) to 4 (university degree). 

Aged 16 and over Dummy variable where 1= Aged 16 or more on 1st January 

1994. 

Ability VRNA, combined verbal reasoning and numerical ability 

scores; centred on its mean value. 

Streaming Extent of streaming and associated curricular differentiation 

in the school; Guttman scale ranging from 0 (mixed ability 
base classes) to 4 (highly streamed). 

Set of dummy variables where 1= in top, middle or 
bottom/remedial class respectively; contrasted with 
membership of mixed ability base class. 

Disciplinary climate School-level average of pupil rating of school as “strict”- 

”easy-going”. 

Positive teacher interaction Likert scale based on frequency of following items: 

(1) Have you been told that your work is good? 

(2) Have you been asked questions in class? 

(3) Have you been praised for answering a difficult question 
correctly? 

(4) Have you been praised because your written work is well 
done? 

Reliabihty; alpha is 0.68. 

Ranges from 0 (low) to 3 (high). 



Top class 
Middle class 
Bottom class 
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Negative teacher interaction 



Academic climate 



ERIC 



Likert scale based on frequency of following items; 

(1) Have you been given out to because your work is untidy 
or not done on time? 

(2) Have you wanted to ask or answer questions in class but 
were ignored? 

(3) Have you been given out to for misbehaving in class? 

(4) Teachers pay more attention in class to what some pupils 
say than to others. 

(5) 1 find most teachers hard to talk to. 

Reliability; alpha is 0.59 (JC) and 0.61 (LC). 

Values range from 0 (low) to 3 (high). 

School-level average of the highest qualification which 
teachers expect the pupil to get. Ranges from 1 (Junior Cert) 
to 4 (University Degree). 
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